Skip to content

Support dots in target names when writing to an override block - #6148

Merged
ilyakuz-db merged 1 commit into
configsync/split-list-write-backfrom
configsync/dotted-target-name
Aug 3, 2026
Merged

Support dots in target names when writing to an override block#6148
ilyakuz-db merged 1 commit into
configsync/split-list-write-backfrom
configsync/dotted-target-name

Conversation

@ilyakuz-db

@ilyakuz-db ilyakuz-db commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Stacked on #6138. Based on its branch so this diff shows only the target-name fix; rebase onto main once #6138 merges.

Problem

A target name may contain a dot — targets: {dev.eu: ...} passes bundle validate and deploys. But write-back addresses an override block by building targets.<target>.<field> as a string, which is then parsed back into path segments. A dotted name splits into two keys, so the write targets /targets/dev/eu/... and the command fails outright:

field not found in YAML configuration: op replace /targets/dev/eu/...: parent path /targets/dev does not exist

Pre-existing, not introduced by #6138: main fails the same way with a different message.

Solution

Build the prefix as path nodes instead of concatenating text. structpath.NewPatternStringKey already switches to bracket notation for a name that is not a plain field, so dev.eu renders as targets['dev.eu'] and round-trips as one key. Both the routed path and the unrouted fallback go through the same helper.

Tests

split/dotted_target edits a field in the override block of a target named dev.eu. It fails on both engines without this change (non-zero exit, nothing written) and passes with it. No other golden changes.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

4 files changed
Suggested: @pietern
Also eligible: @denik, @andrewnester, @anton-107, @janniklasrose, @lennartkats-db, @shreyas-goenka

/bundle/ - needs approval

Files: bundle/configsync/blockindex.go, bundle/configsync/resolve.go
Suggested: @pietern
Also eligible: @denik, @andrewnester, @anton-107, @janniklasrose, @lennartkats-db, @shreyas-goenka

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 1544e89

Run: 30845546883

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 334 1078 9:08
💚​ aws windows 4 4 324 1080 8:02
💚​ azure linux 4 4 334 1077 9:38
💚​ azure windows 4 4 324 1079 8:15
💚​ gcp linux 1 5 333 1079 10:55
💚​ gcp windows 1 5 323 1081 8:09
8 interesting tests: 4 RECOVERED, 4 SKIP
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 9 slowest tests (at least 2 minutes):
duration env testname
6:33 gcp windows TestAccept
6:20 aws windows TestAccept
6:10 azure windows TestAccept
3:00 azure linux TestAccept
2:54 gcp linux TestAccept
2:48 aws linux TestAccept
2:31 gcp windows TestFilerReadWrite/workspace_files
2:22 azure linux TestAccept/bundle/config-remote-sync/job_multiple_tasks/DATABRICKS_BUNDLE_ENGINE=terraform
2:04 azure linux TestAccept/bundle/config-remote-sync/job_multiple_tasks/DATABRICKS_BUNDLE_ENGINE=direct

@ilyakuz-db
ilyakuz-db force-pushed the configsync/dotted-target-name branch from 03fd794 to 04c769a Compare August 3, 2026 15:43
@ilyakuz-db
ilyakuz-db force-pushed the configsync/split-list-write-back branch from 49300b3 to 8b3e0a8 Compare August 3, 2026 16:04
@ilyakuz-db
ilyakuz-db force-pushed the configsync/dotted-target-name branch 3 times, most recently from 0df0b5d to b4c479d Compare August 3, 2026 16:32
@ilyakuz-db
ilyakuz-db force-pushed the configsync/dotted-target-name branch 2 times, most recently from 71e1909 to 8b3fd57 Compare August 3, 2026 17:26
@ilyakuz-db
ilyakuz-db force-pushed the configsync/dotted-target-name branch from 8b3fd57 to 1544e89 Compare August 3, 2026 19:21
@ilyakuz-db
ilyakuz-db merged commit 1544e89 into configsync/split-list-write-back Aug 3, 2026
21 checks passed
@ilyakuz-db
ilyakuz-db deleted the configsync/dotted-target-name branch August 3, 2026 22:08
yolocs pushed a commit to yolocs/dbcli that referenced this pull request Aug 4, 2026
…hange (databricks#6138)

## Problem

A resource's sequence field (e.g. job `tasks`) can be written in two
physical YAML regions: the top-level `resources.*` block and a
`targets.` override, either possibly in its own included file. Loading
merges them into one list and sorts keyed lists by key.

Config-sync resolves a change against that **merged** list, then writes
it using the merged position, which causes out-of-bound indexing issues
or issues with config corruption

## Solution

`bundle/configsync/blockindex.go` maps every `dyn.Location` back to the
block that owns it (not only a file), so a destination is *looked up*,
and `resolveSelectors` rewrites merged positions into block-local ones.

Blocks are recovered by re-parsing the contributing files, because
selecting a target destroys the distinction (it folds overrides into
`resources` and drops `targets`). Re-parsing, rather than reloading via
the mutator pipeline, avoids running the bundle's `preinit` script twice
per sync.

Guiding rule: **correct write > no write > wrong write.** Anything
ambiguous is left for a later run.

How a destination is picked:

- element defined in one block → that block;
- element defined in several → the **field** decides, since each field
has its own location, so two fields of one task can go to two different
blocks;
- field itself defined in several → the definition that **won the
merge**: the target override, or between two files in the same scope the
one loaded later. Writing the shadowed copy would leave the effective
value unchanged, so the sync would re-detect the same change on every
run;
- **removing** anything defined in several blocks → one deletion per
definition, since it only disappears once every copy is gone. This
applies to a *field* as well as an element: deleting only the winning
copy lets the shadowed one take effect, so the next deploy restores the
value the user just removed;
- **renaming** → a replace of just the key field in every defining
block, so other fields stay in their own scope. A key change arrives as
an unlinked remove plus add; the two are paired by identity;
- brand-new element or field → the block declaring the resource. A field
a mutator inserted (e.g. `OverrideCompute` setting
`existing_cluster_id`) counts as new: the value is present but carries
no location, so no block declares it;
- otherwise → left unapplied rather than written to a guess.

Also worth knowing: index bookkeeping is per block, so a removal in one
block cannot shift positions in another; rename pairing refuses
ambiguous matches in both directions; and a target override may
legitimately span several files, so a block is identified by (scope,
file)

## Tests

New acceptance tests, a lot of test cases to cover everything that this
refactoring touched, but all are local as we only need to cover config
resolution

## Follow-ups

Stacked on this branch, each independently reviewable:

- **databricks#6146** — report changes that were detected but not written back.
Today every drop path is a `log.Debugf` while the command prints the
full changeset and exits 0, so a dropped edit is indistinguishable from
an applied one.
- **databricks#6148** — support a dot in a target name (`targets: {dev.eu: ...}`).
Pre-existing on `main`, but it fails the whole command rather than
skipping one change.

Rebase onto `main` once this merges.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants